<html>
<head><meta charset="utf-8"><title>Masking pointer without losing provenance · t-lang/wg-unsafe-code-guidelines · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/index.html">t-lang/wg-unsafe-code-guidelines</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html">Masking pointer without losing provenance</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="216083638"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216083638" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Diggsey <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216083638">(Nov 09 2020 at 13:33)</a>:</h4>
<p>What is the most efficient way to mask off the lower bits of a pointer without inhibiting any optimizations now or in the future?</p>



<a name="216101904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216101904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jacob Lifshay <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216101904">(Nov 09 2020 at 15:45)</a>:</h4>
<p><a href="https://play.rust-lang.org/?version=stable&amp;mode=release&amp;edition=2018&amp;gist=63d90209f533c2f878c282494226eb70">https://play.rust-lang.org/?version=stable&amp;mode=release&amp;edition=2018&amp;gist=63d90209f533c2f878c282494226eb70</a><br>
seems to work for me, it shouldn't lose provenance</p>



<a name="216102004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216102004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jacob Lifshay <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216102004">(Nov 09 2020 at 15:46)</a>:</h4>
<p>on x86, it optimizes to just an <code>and</code> instruction</p>



<a name="216110246"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216110246" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Diggsey <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216110246">(Nov 09 2020 at 16:40)</a>:</h4>
<p>Interesting - I tried that first on godbolt, and it did not optimize it correctly. After seeing it work on the playground I realised that its because godbolt defaults to an older rust version</p>



<a name="216110334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216110334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Diggsey <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216110334">(Nov 09 2020 at 16:40)</a>:</h4>
<p>Specifically, in Rust 1.44, this code is not optimized</p>



<a name="216130964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216130964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> scottmcm <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216130964">(Nov 09 2020 at 19:17)</a>:</h4>
<p>Hmm, there's <a href="https://doc.rust-lang.org/std/primitive.pointer.html#method.align_offset">https://doc.rust-lang.org/std/primitive.pointer.html#method.align_offset</a> for round-up, but I don't know what round-down would be.  Could try using that with some checks and seeing whether it optimizes away...</p>



<a name="216138489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216138489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jacob Lifshay <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216138489">(Nov 09 2020 at 20:22)</a>:</h4>
<p>Clang just seems to use the C++ equivalent of <code>(ptr as usize &amp; !3usize) as *mut T</code>.</p>
<p>Random example: <a href="https://github.com/llvm/llvm-project/blob/2c2680a470b59df84db16c4cd876efce91d7d4f2/clang/include/clang/Sema/IdentifierResolver.h#L195">https://github.com/llvm/llvm-project/blob/2c2680a470b59df84db16c4cd876efce91d7d4f2/clang/include/clang/Sema/IdentifierResolver.h#L195</a></p>



<a name="216729131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216729131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216729131">(Nov 14 2020 at 13:31)</a>:</h4>
<p>to answer to the question that is given in the issue title... there's no good way in Rust (or any other language that I know of)</p>



<a name="216729155"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216729155" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216729155">(Nov 14 2020 at 13:31)</a>:</h4>
<p>but you can do weird things like</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="sd">/// Return a pointer with the address of `ptr` and the provenance of `prov`.</span>
<span class="k">fn</span> <span class="nf">copy_provenance</span><span class="p">(</span><span class="n">ptr</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">i8</span><span class="p">,</span><span class="w"> </span><span class="n">prov</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">i8</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">i8</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">  </span><span class="kd">let</span><span class="w"> </span><span class="n">diff</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">ptr</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">isize</span><span class="p">).</span><span class="n">wrapping_sub</span><span class="p">(</span><span class="n">prov</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">isize</span><span class="p">);</span><span class="w"></span>
<span class="w">  </span><span class="n">prov</span><span class="p">.</span><span class="n">wrapping_offset</span><span class="p">(</span><span class="n">diff</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>(I hope I got this right^^)</p>



<a name="216729390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216729390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216729390">(Nov 14 2020 at 13:35)</a>:</h4>
<p>then you can do your masking, which <em>will</em> lose provenance, and at the end re-add the original provenance</p>



<a name="216729399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216729399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216729399">(Nov 14 2020 at 13:35)</a>:</h4>
<p>that said, I do not know of any compiler where this would actually help to get more optimizations^^</p>



<a name="216729452"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216729452" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216729452">(Nov 14 2020 at 13:36)</a>:</h4>
<p>(but it could help to fix soundness issues caused by incorrect provenance)</p>



<a name="216731848"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216731848" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216731848">(Nov 14 2020 at 14:30)</a>:</h4>
<p>I then wonder if a <code>.map_addr()</code> convenient adapter method could help people do <del>the right thing</del> a less bad thing:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">ptr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ptr</span><span class="p">.</span><span class="n">map_addr</span><span class="p">(</span><span class="o">|</span><span class="n">addr</span><span class="o">|</span><span class="w"> </span><span class="n">addr</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="n">MASK</span><span class="p">);</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>with:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">impl</span><span class="w"> </span><span class="o">*</span><span class="n">T</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">unsafe</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">map_addr</span><span class="w"> </span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f</span>: <span class="nc">impl</span><span class="w"> </span><span class="nb">FnOnce</span><span class="p">(</span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="kt">usize</span><span class="p">)</span><span class="w"></span>
<span class="w">      </span>-&gt; <span class="nc">Self</span><span class="w"></span>
<span class="w">    </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">prov</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">;</span><span class="w"></span>
<span class="w">        </span><span class="n">copy_provenance</span><span class="p">(</span><span class="n">f</span><span class="p">(</span><span class="bp">self</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="bp">Self</span><span class="p">,</span><span class="w"> </span><span class="n">prov</span><span class="p">)</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Or to have a <code>.raw_split(): *T -&gt; (usize, Provenance)</code> method w/ a <code>Provenance::imbue(addr: usize)</code> pointer constructor <span aria-label="thinking" class="emoji emoji-1f914" role="img" title="thinking">:thinking:</span></p>



<a name="216732113"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216732113" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216732113">(Nov 14 2020 at 14:36)</a>:</h4>
<p>This would solve the "second issue" that the slice concatenating methods have. That is, IIUC, the following breaks provenance:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="k">ref</span><span class="w"> </span><span class="n">alloc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">42</span><span class="p">,</span><span class="w"> </span><span class="mi">27</span><span class="p">];</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">slice</span>: <span class="kp">&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="n">alloc</span><span class="p">[</span><span class="o">..</span><span class="p">];</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">hd</span><span class="p">,</span><span class="w"> </span><span class="n">tl</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slice</span><span class="p">.</span><span class="n">split_at</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">slice</span>: <span class="kp">&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">slice_concat</span><span class="p">(</span><span class="n">hd</span><span class="p">,</span><span class="w"> </span><span class="n">tl</span><span class="p">)</span><span class="w"> </span><span class="p">};</span><span class="w"> </span><span class="c1">// Safety: originate from same alloc</span>
</code></pre></div>
<p>does it not?</p>



<a name="216733696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216733696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216733696">(Nov 14 2020 at 15:05)</a>:</h4>
<p>for slice concat, <em>there is no correct provenance</em> as each provenance pins the ptr to a particular allocation</p>



<a name="216733697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216733697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216733697">(Nov 14 2020 at 15:05)</a>:</h4>
<p>so I dont think this can help there</p>



<a name="216733752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking%20pointer%20without%20losing%20provenance/near/216733752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/136281-t-lang/wg-unsafe-code-guidelines/topic/Masking.20pointer.20without.20losing.20provenance.html#216733752">(Nov 14 2020 at 15:07)</a>:</h4>
<p>also I am not sure if there is anything "bad" about the usual <code>((addr as usize) &amp; mask) as *mut _</code>. it loses provenance, okay... so what? from all I know, this code still <em>has to</em> work. don't get me wrong, if we could make this code illegal that would solve a metric ton of problems, but also I'd be shunned out of the Rust community for breaking everything.^^</p>
<p>this is exactly what makes int-to-ptr casts so hard, that something needs to happen with provenance. but I doubt just declaring them illegal is a viable option. unfortunately.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>